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OVERVIEW 


The  LBNL  group  has  developed  a  prototype  of  a  hand-held  UXO  discriminator  (14-in  =  0.35  m 
cube)  that  is  able  to  discriminate  small  (20  mm)  UXO  objects  at  a  depth  of  0.45  m  and  large  (105 
and  155  mm)  UXO  objects  at  the  depth  of  0.85  m  from  harmless  scrap  metal.  This  hand-held 
prototype  incorporates  the  key  features  of  the  cart-mounted  system:  three  orthogonal  transmitters 
and  ten  pairs  of  receivers,  and  difference  or  gradient  measurements  that  significantly  reduce  the 
ambient  and  motion  noise,  and  greatly  enhance  the  sensitivity  to  the  gradients  of  the  target.  The 
system  characterizes  the  target  from  a  single  position.  A  demonstration  survey  at  a  standard  test 
site  showed  that  the  same  discrimination  capabilities  afforded  by  the  cart-mounted  system  are 
available  in  the  hand-held  unit,  although  with  a  slightly  reduced  depth  of  detection.  To 
demonstrate  the  system  performance  at  a  live  site,  LBNL  participated  in  the  ESTCP  UXO 
discrimination  study  at  the  former  Camp  Beale,  Marysville,  CA  where  the  hand-held  UXO 
discriminator  was  operated  in  the  cued  mode.  The  system  was  brought  to  marked  locations 
(flags)  and  ran  in  the  characterization/discrimination  mode.  The  system  had  no  positioning 
capability;  hence  data  around  the  flag  were  acquired  using  a  three  or  five-point  template  with  the 
system  oriented  in  a  single  direction.  The  survey  took  two  weeks,  and  we  occupied  900  flag 
locations  within  18  grids  selected  by  the  program  office.  UXO  and  scrap  discrimination  and 
priority  dig  list  were  be  done  by  Signal  Innovations  Group,  Inc. 
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1.  INTRODUCTION 


In  2003,  the  Defense  Science  Board  observed:  “The  ...  problem  is  that  instruments  that  can 
detect  the  buried  UXOs  also  detect  numerous  scrap  metal  objects  and  other  artifacts,  which  leads 
to  an  enormous  amount  of  expensive  digging.  Typically  100  holes  may  be  dug  before  a  real 
UXO  is  unearthed!  The  Task  Force  assessment  is  that  much  of  this  wasteful  digging  can  be 
eliminated  by  the  use  of  more  advanced  technology  instruments  that  exploit  modem  digital 
processing  and  advanced  multi-mode  sensors  to  achieve  an  improved  level  of  discrimination  of 
scrap  from  UXOs.” 

With  prior  funding  (UX-1225,  MM-0437,  and  MM-0838),  LBNL  group  has  successfully 
designed  and  built  a  cart-mounted  Berkeley  UXO  Discriminator  (BUD)  and  demonstrated  its 
performance  at  various  test  sites  (e.g.,  Gasperikova  et  al.,  2007,  2008,  and  2009). 

Hand-held  systems  have  the  advantage  of  being  lightweight,  compact,  portable,  and  deployable 
under  most  site  conditions.  They  are  particularly  useful  in  areas  of  dense  vegetation  or 
challenging  terrain.  In  heavily  wooded  areas  or  areas  with  steep  or  uneven  terrain,  hand-held 
sensors  may  be  the  only  suitable  sensor  deployment  method.  A  useful  criterion  might  be  that  it 
could  be  carried  through  spaces  that  the  operator  could  walk  through  or  at  least  approach. 
Further,  it  is  desirable  to  find  and  characterize  a  metallic  object  without  the  need  to  accurately 
locate  the  sensors  at  multiple  positions  around  the  target.  The  ideal  system  would  thus  locate  and 
characterize  the  target  from  a  single  position  of  the  sensor  and  indicate  to  the  operator  where  to 
flag  the  target  for  subsequent  study. 
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A  sensor  package  of  a  14-in  (0.35  m)  cube  was  designed  and  built  based  on  these  considerations 
(UX-1667).  This  hand-held  prototype  incorporates  the  key  features  of  the  cart-mounted  system  - 
(a)  three  orthogonal  transmitters  and  ten  pairs  of  receivers,  and  (b)  difference  or  gradient 
measurements  that  significantly  reduce  the  ambient  and  motion  noise,  and  greatly  enhance  the 
sensitivity  to  the  gradients  of  the  target.  The  system  characterizes  the  target  from  a  single 
position.  Results  from  a  local  test  site  and  the  Aberdeen  Proving  Ground  (APG)  were  in  a  good 
agreement  with  theoretical  performance  calculations  of  such  a  device  (Gasperikova  et  al.  2010). 
This  survey  was  designed  to  demonstrate  the  system  performance  under  realistic  survey 
conditions  at  the  former  Camp  Beale,  Marysville,  CA. 
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2.  TECHNOLOGY  DESCRIPTION 


The  SERDP  and  ESTCP  have  supported  LBNL  in  development  of  a  device  that  not  only  detects 
the  object  itself  but  also  quantitatively  determines  its  size,  shape,  and  orientation.  In  a  hand-held 
system  the  sensor  package  must  be  easily  maneuvered  over  rough  terrain,  through  brush,  etc.  A 
useful  criterion  might  be  that  it  could  be  carried  through  spaces  that  the  operator  could  walk 
through  or  at  least  approach.  To  avoid  the  reliance  on  accurate  multiple  positioning  of  any 
system  it  has  been  shown  that  (1)  the  object  must  be  illuminated  with  three  different 
polarizabilities  of  transmitted  field  and  that  (2)  to  determine  location  from  a  single  position  of 
the  transmitter-receiver  system  multiple  spaced-apart  receivers  must  be  used.  To  accommodate 
the  first  requirement  a  hand-held  design  implements  three  orthogonal  transmitter  loops  much  like 
the  cart-mounted  system.  To  accommodate  the  second  requirement  the  hand-held  UXO 
discriminator  uses  ten  pairs  of  receivers.  The  same  discrimination  abilities  afforded  by  the  cart- 
mounted  system  are  available  in  the  hand-held  unit  albeit  with  slightly  reduced  depth  of 
detection.  The  hand-held  UXO  discriminator  is  able  to  discriminate  small  (20  mm)  objects  at  a 
depth  of  0.45  m  and  large  (105  and  155  mm)  objects  at  the  depth  of  0.85  m  and  detect  them 
down  to  1.15  m. 

The  hand-held  UXO  discriminator  employs  three  orthogonal  transmitters  and  ten  pairs  of 
differenced  receivers.  Each  vertical  face  of  the  cube  has  three  induction  coils,  and  two  horizontal 
faces  have  four  induction  coils,  each  sensitive  to  the  magnetic  field  component  normal  to  the 
face.  Receivers  on  opposite  faces  of  the  cube  are  paired  along  the  symmetry  lines  through  the 
center  of  the  system  and  each  pair  sees  identical  fields  during  the  on-time  of  current  pulses  in  the 
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transmitter  coils.  They  are  wired  in  opposition  to  produce  zero  output  during  the  on-time  of  the 
pulses  in  three  orthogonal  transmitters.  This  configuration  dramatically  reduces  noise  in 
measurements  by  canceling  background  electromagnetic  fields  (these  fields  are  uniform  over  the 
scale  of  the  receiver  array  and  are  consequently  nulled  by  the  differencing  operation),  and  by 
canceling  noise  contributed  by  the  tilt  of  the  receivers  in  the  Earth’s  magnetic  field,  thus  greatly 
enhances  receivers’  sensitivity  to  gradients  of  the  target  response.  The  hand-held  UXO 
discriminator  (14-in  or  0.35  m  cube),  shown  in  Figure  1,  is  able  to  discriminate  small  (20  mm) 
objects  at  a  depth  of  0.45  m  and  large  (105  and  155  mm)  objects  at  the  depth  of  0.85  m  and 
detect  them  down  to  1.15  m.  Transmitter-receiver  configuration  is  shown  in  Figure  2a  while  the 
assembled  transmitter-receiver  cube  is  shown  in  Figure  2b. 


Figure  1.  FBNF  hand-held  UXO  discriminator 
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Figure  2.  LBNL  hand-held  UXO  discriminator:  (a)  transmitter-receiver  configuration,  (b) 
assembled  prototype. 


Data  acquisition  is  performed  on  a  single  board.  The  transmitter  coils  are  powered  separately 
from  the  data  acquisition  board.  Pulsers  provide  resonant  circuit  switching  to  create  bi-polar  half¬ 
sine  pulses  of  300  us  width.  The  current  has  a  -40  A  peak  and  a  resonant  circuit  voltage  of  -400 
Volts,  and  the  operational  overall  half-sine  duty  cycle  is  -12%.  Transients  are  digitized  with  a 
sampling  interval  of  4  us.  The  sensors  are  critically  damped  5-inch  coils  with  a  self-resonant 
frequency  of  75  kHz.  The  data  acquisition  board  has  12  high-speed  ADC  channels  for  output. 
Ten  of  these  channels  are  used  for  the  signal  from  receiver  coils,  and  the  remaining  two  channels 
provide  information  about  the  system  (i.e.  tilt  information,  transmitter  current). 

An  important  feature  of  the  hand-held  UXO  discriminator  is  an  inversion  algorithm,  which  is 
used  to  determine  target  properties  from  measurements  with  a  given  transmitter-receiver 
configuration.  At  any  given  time  the  response  is  inverted  to  yield  the  location  (x,  y,  z)  of  the 
target,  its  attitude  and  its  principal  polarizabilities.  It  has  been  demonstrated  that  a  satisfactory 
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classification  scheme  is  one  that  determines  the  principal  dipole  polarizabilities  of  a  target  -  a 
near  intact  UXO  displays  a  single  major  polarizability  coincident  with  the  long  axis  of  the  object 
and  two  equal  transverse  polarizabilities.  Figures  3  and  4  illustrate  a  discrimination  capability  of 
the  system  for  UXO  object  and  a  scrap  metal,  respectively.  While  UXO  objects  have  a  single 
major  polarizability  coincident  with  the  long  axis  of  the  object  and  two  equal  transverse 
polarizabilities  (Figure  3),  scrap  metal  exhibits  three  distinct  principal  polarizabilities  (Figure  4). 
There  are  clear  distinctions  between  symmetric  intact  UXO  and  irregular  scrap  metal.  Moreover, 
UXO  have  unique  polarizability  signatures,  and  thus  distinctions  can  be  made  among  various 
UXOs.  Object  orientation  and  equivalent  dipole  polarizability  estimates  of  large  and  shallow 
UXO  or  scrap  are  more  problematic  as  they  are  affected  by  higher  order  (non-dipole)  terms 
induced  in  objects  due  to  source  field  gradients  along  the  length  of  the  objects.  In  the  case  of  the 
large  and  shallow  objects,  the  hand-held  system  can  be  easily  raised  an  appropriate  distance 
above  the  object  such  that  the  dipole  model  approximation  for  polarizability  estimates  is 
appropriate. 

UXO  and  scrap  discrimination  and  priority  dig  list  were  done  by  Signal  Innovations  Group,  Inc. 
and  the  approach  is  described  in  Chapter  4.2. 
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Figure  3.  Inversion  results  for  the  principal  polarizabilities  of  81  mm  mortar. 


0.0001 


0.001 
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Figure  4.  Inversion  results  for  the  principal  polarizabilities  of  19x8  cm  scrap  metal. 
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3.  DEMONSTRATION  SURVEY 


The  demonstration  survey  was  performed  between  June  13  and  June  28,  2011.  The  hand-held 
UXO  discriminator  was  operated  in  the  cued  mode.  The  system  was  brought  to  marked  locations 
(flags)  and  ran  in  the  characterization/discrimination  mode.  The  three  discriminating 
polarizability  responses  along  with  the  object  depth  and  horizontal  location  with  respect  to  the 
center  of  the  bottom  plane  of  the  cube  was  recorded  and  visually  presented  on  the  computer 
screen.  Additional  values  recorded  with  each  location  were  S/N  ratio,  polarizability  index  (an 
average  value  of  the  product  of  time  (in  seconds)  and  polarizability  rate  (in  m  /s)  over  the  46 
sample  times  logarithmically  spaced  from  80  to  1,460  p,s),  and  the  cart  orientation,  pitch,  and 
roll.  Figure  5  illustrates  the  deployment  of  the  system  at  the  site. 


Figure  5.  LBNL  hand-held  UXO  discriminator  at  Camp  Beale. 
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Soundings  were  collected  at  each  flag,  and  at  0.15  m  before  and  0.15  m  after  it,  with  the  system 
oriented  in  a  single  direction.  Soundings  were  differenced  with  background  reference  soundings 
taken  within  the  previous  30-40  minutes  at  a  nearby  site  determined  by  the  field  operator  to  be 
free  from  metallic  objects.  When  polarizability  inversions  from  all  three  soundings  were 
unrelated  to  each  other,  additional  two  soundings  were  taken,  one  on  each  side  of  the  system, 
0.15  m  from  the  flag.  The  hand-held  UXO  discriminator  surveyed  18  grids  that  are  indicated  in 
Figure  6  in  yellow  and  blue  colors.  These  grids  were:  J02,  J03,  J05,  J06,  K02,  K03,  K05,  K06, 
R07-09,  S02,  S03,  S07-09,  T02,  and  T03. 


UTM  Easting  (m) 


|__|  Site  Boundary 
□  Portable  Grids 
I  I  Combined  Grids 

I  I  MetalMapper  Grids 


Figure  6.  Survey  area  divided  into  sub-grids. 
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4.  DATA  ANALYSIS  AND  INTERPRETATION 


4.1  Data  Processing 

Ten  channels  of  field  data  were  recorded  at  a  rate  of  250  k-samples/second  for  each  of  three 
transmitters.  Field  data  were  stacked  together  in  a  field  programmable  gate  array  (FPGA)  and 
transferred  to  a  field  computer  (laptop)  forming  a  primitive  stack.  An  even  number  of  primitive 
stacks  is  averaged  together  to  form  stacked  data  for  further  processing.  The  peak  transmitter 
current  was  estimated  from  the  stacked  transmitter  current  channel  record,  and  the  data  were 
normalized  by  that  value.  Nominal  transmitter  shut-off  time  was  estimated,  and  induction 
responses  were  computed  at  46  logarithmically  spaced  times  between  80  and  1,460  ps,  averaged 
in  half-sine  windows  with  widths  10%  the  center  time  after  transmitter  pulse  shut-off.  Responses 
were  differenced  with  background  responses  collected  over  an  area  determined  to  be  free  of 
metallic  objects.  The  resulting  30  channels  of  normalized  responses  were  then  inverted  for 
candidate  object  position  and  principal  polarizabilities  as  a  function  of  time  after  transmitter 
shut-off.  These  were  used  as  an  input  for  the  discrimination  approaches  described  in  Chapters  4.2 
and  4.3. 


4.2  Discrimination  Using  the  Isolate  Process 

Signal  Inovations  Group,  Inc.  (SIG)  applied  the  Isolate  discrimination  process  in  the  Camp  Beale 
demonstration.  This  process  involved  robust  feature  extraction  and  selection  and  semi-supervised 
classifier  training  using  active  selection  of  labeled  data.  As  shown  in  Figure  7,  this  process  was 


Camp  Beale  Demonstration  Report 


14 


April  2012 


divided  into  three  main  phases:  (1)  Feature  Extraction,  (2)  Site  Learning,  and  (3)  Excavation. 
Details  on  each  phase  are  given  in  the  next  subsections.  The  SIG  Isolate  process  leveraged  on 
two  feedback  steps.  The  first  feedback  was  in  training  the  semi-supervised  classifier,  where 
additional  anomaly  labels  were  requested  until  the  classifier  reached  sufficient  stability.  The 
second  feedback  was  during  the  excavation  of  anomalies,  where  the  classifier  was  retrained  with 
additional  labeled  anomalies  until  either  the  UXO/clutter  predictions  became  highly  separable  or 
until  high  probability  anomalies  were  substantially  revealed  to  be  clutter  upon  excavation. 


FEATURE  SITE  LEARNING  EXCAVATION 

EXTRACTION  PHASE  PHASE 

PHASE 


Figure  7.  Flow  diagram  of  the  SIG  Isolate  process. 
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4.2.1  Feature  Selection  with  BENet 


Adaptive  learning  of  a  classifier  in  situ  benefits  from  refining  the  appropriate  set  of  extracted 
features  for  the  targets  under  test.  This  occurs  because  of  the  ‘curse  of  dimensionality’  where  the 
number  of  data  points  required  to  cover  the  breadth  of  a  features  space  grows  exponentially  with 
the  number  of  features  considered.  If  the  amount  of  training  data  does  not  sufficiently  sample  the 
feature  space,  then  the  learned  classifier  will  lack  statistical  support  and  class  estimate 
uncertainty  is  large.  Bayesian  classification  models  perform  feature  selection  by  placing  a 
sparseness  prior  on  the  inferred  feature  weights.  The  Bayesian  elastic  net  (BENet)  regression 
model  used  for  feature  selection  employs  a  sparseness  prior  equivalent  to  a  convex  combination 
of  LI -norm  and  L2-norm  penalties  in  a  least  squares  optimization  formulation  (Zhang  et  al., 
2003;  Zhang  et  al.,  2004).  The  sparseness  prior  of  the  BENet  model  jointly  infers  the  essential 
subset  of  relevant  features,  including  correlated  features,  for  a  given  classification  task.  Rather 
than  encouraging  the  selection  of  a  single  feature  in  a  set  of  correlated  important  features  (like 
similar  approaches  such  as  Relevance  Vector  Machine),  the  BENet  model  encourages  the 
selection  of  all  correlated  important  features.  By  performing  sparse  and  grouped  feature 
selection,  the  BENet  algorithm  provides  a  more  robust  approach  to  feature  adaptability  and  the 
interpretation  of  important  features,  ultimately  requiring  fewer  training  data  samples  to  achieve 
robust  statistical  support. 
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4.2.2  Semi-Supervised  Learning  Classifier 


Semi-supervised  learning  is  applicable  to  any  sensing  problem  for  which  all  of  the  labeled  and 
unlabeled  data  are  available  at  the  same  time,  and  therefore,  particularly  for  the  current 
demonstration  study.  In  most  practical  applications,  semi-supervised  learning  has  been  found  to 
yield  superior  performance  relative  to  widely  applied  supervised  algorithms,  which  train  only  on 
labeled  data.  The  context  provided  by  the  unlabeled  data  is  crucial  in  improving  the  classification 
performance  especially  in  cases  where  the  labeled  data  were  not  representative  of  the  two  class 
distributions.  As  the  number  of  training  samples  increases,  the  supervised  classifier  should 
approximate  the  semi-supervised  classifier.  Semi-supervised  formulation  treats  the  dataset 
(labeled  and  unlabeled)  as  a  set  of  connected  nodes,  where  the  affinity  between  any  two 
feature  vectors  (nodes)  and  f  j  is  defined  in  terms  of  a  radial  basis  function  (Cristianini  and 
Shawe-Taylor,  2000).  Based  on  the  above  formulation,  one  can  design  a  Markov  transition 
matrix  A  =  [dij]  that  represents  the  probability  of  transitioning  from  node  fl  to  /) . 

Assuming  L  Q  (1,2, ... ,  NL}  represents  the  set  of  labeled  data  indices,  the  likelihood  functional 
can  be  written  as 

Ni 

({ y%,i  e  L}\Jsr(fi),0)  =  J^[p(y;|.Af(/U0)  =  nx  aijp{yi\fj,0) 

iEL  iEL  j- 1 

where  JSf  (/)  defines  the  neighborhood  of  / .  Estimation  of  classifier  parameters  0  can  be 
achieved  by  maximizing  the  log-likelihood  via  an  Expectation-Maximization  algorithm  (Liao 
and  Carin,  2008).  To  enforce  sparseness  of  0  (enforcing  most  of  the  components  of  the 
parameter  vector  0  to  be  zero),  one  may  impose  a  zero-mean  Gaussian  prior  on  0.  A  zero-mean 
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Gaussian  prior  with  appropriate  variance  can  strongly  bias  the  algorithm  in  choosing  parameter 
weights  that  are  most  likely  very  small  (close  to  zero). 

4.2.3  Batch-mode  Active  Learning  (AL) 

Given  that  available  training  data  labels  at  the  beginning  of  a  demonstration  are  not  available  and 
that  excavations  must  be  performed  to  reveal  training  data  labels,  one  may  ask  in  which  order 
anomalies  should  be  excavated  to  maximally  improve  the  performance  of  the  classifier 
algorithm.  One  useful  criterion  is  to  use  the  confidence  on  the  estimated  identity  of  the  anomalies 
that  are  yet  to  be  excavated.  Specifically,  one  may  ask  which  unlabeled  anomaly  label  would  be 
most  informative  to  improve  classifier  performance  if  the  associated  label  could  be  made 
available.  It  has  been  shown  by  MacKay  (1992)  that  this  question  can  be  answered  in  a 
quantitative  information-theoretic  manner. 

For  active  label  selection,  posterior  distribution  of  the  classifier  is  approximated  as  a  Gaussian 
distribution  centered  on  the  maximum  a  posteriori  (MAP)  estimate.  The  uncertainty  of  the 
classifier  is  quantified  in  terms  of  the  posterior  precision  matrix.  The  objective  of  AL  is  to  choose 
a  feature  vector  for  labeling  that  maximizes  the  mutual  information  (/)  between  the  classifier  0 
and  the  new  data  point  to  be  labeled.  The  mutual  information  can  be  quantified  as  the  expected 
decrease  of  the  entropy  of  0  after  new  sample  / j*  and  its  label  yit  are  observed. 

7  =  =  +  P(yi*l/«*,0)x[l  -p(yi*l/i*,0)]/r*H-1/i*} 

It  is  important  to  note  that  the  mutual  information  I  is  large  when  p(y **  \xit,  0 )  ~  0.5.  Hence,  the 
AL  prefers  to  acquire  labels  on  those  unlabeled  samples  for  which  the  current  classifier  is  most 
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confused  or  uncertain.  In  this  fashion  the  classifier  learns  quickly  by  not  excavating  anomalies 
that  reveal  redundant  information.  The  process  continues  as  new  labels  are  revealed  until  the 
expected  information  gain  for  the  remaining  anomalies  is  approximately  uniformly  low.  At  that 
point  the  classifier  is  adequately  trained  and  target  inference  on  the  remaining  unlabeled 
anomalies  can  be  reliably  performed.  By  invoking  the  principle  of  submodularity  in  the 
algorithm  optimization,  the  approach  has  been  adapted  to  allow  for  the  selection  of  multiple 
simultaneous  labels  at  one  time,  making  the  technique  operationally  practical. 

4.2. 4  SIG  Isolate  Process  Overview 

The  SIG  Isolate  process  can  be  summarized  in  the  following  steps  (Figure  7): 

•  Data  Conditioning  -  First,  raw,  unlabeled  anomaly  data  are  received. 

•  Subspace  Denoising  -  The  anomaly  data  are  denoised  to  ensure  robust  performance  for 
discriminating  late  time-gate  features. 

•  Feature  Extraction  -  A  robust  multi-anomaly  dipole  model  is  fitted  to  the  data.  The 
polarizability  parameters  from  this  fitting  become  the  set  from  which  features  are  drawn 
for  classifier  training.  In  addition  to  the  time-domain  polarizabilities,  a  set  of  9  ‘rate’ 
features  were  calculated.  These  features  were  calculated  by  fitting  the  time-domain 
polarizabilities  of  each  axis  to  an  exponential-decay  model: 

-t 

Pi  =  ru  +  r2iersi 

where  i  E  { x,y,zj  is  the  current  axis,  p  is  the  polarizability,  t  is  time  and  {r1(  r2,  r3}  are 
the  fitted  rate  parameters.  The  optimized  values  of  the  rate  parameters  were  found  using 
non-linear  least  squares. 


Camp  Beale  Demonstration  Report 


19 


April  2012 


•  Basis  Selection  -  A  few  of  the  many  possible  features  are  selected  based  on  their  physical 
interpretation  as  they  relate  to  the  anomaly,  and,  using  these  features,  the  most 
informative  set  of  anomalies  are  selected  via  an  information  metric  to  begin  classifier 
training. 

•  Feature  Set  Augmentation  -  The  feature  set  is  then  augmented  by  adding  early,  mid  and 
late  time  polarizabilities  values. 

•  Automated  Features  Selection  -  The  most  relevant  set  of  features  is  selected  from  the 
larger  feature  set  using  BENet. 

•  Semi-supervised  Parametric  Neighborhood-Based  Classifier  (PNBC)  Training  (STL 
or  MTL)  -  When  the  PNBC  is  trained  only  using  data  from  the  current  site  of  interest,  it 
is  called  Single  Task  Learning  (STL).  When  the  PNBC  is  trained  for  multiple  sites 
simultaneously  it  is  called  MTL. 

•  Batch-mode  Active  Learning  -  Based  on  the  estimates  made  with  the  PNBC  classifier,  a 
new  set  of  anomalies  are  selected  for  labeling  using  batch-mode  AL.  The  goal  at  this  step 
is  to  maximize  the  information  gain  from  new  labels  requested  from  the  set  of  unlabeled 
anomalies.  The  process  is  repeated  as  the  PNBC  classifier  adequately  learns  data 
manifold.  The  stopping  criterion  for  the  learning  process  is  apparent  when  the  remaining 
unlabeled  data  points  have  approximately  equal  information  for  improving  the  classifier, 
and  labeling  any  one  anomaly  is  no  better  than  any  other. 

•  Excavation  Adapted  Threshold  Selection  -  At  this  point,  the  highest  probability  UXO 
are  selected  for  excavation  and  labels.  The  classifier  continues  to  be  retrained  when  new 
labels  are  revealed.  This  process  continues  until  the  highest  probability  UXO  items 
excavated  are  all  found  to  be  clutter  at  which  point  digging  stops. 
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4.2.5  Application  to  Camp  Beale  Data 


The  feature  extraction  phase  started  with  selecting  anomaly  responses  for  the  analysis.  As 
described  in  Chapter  3,  three  or  five  measurements  were  made  around  the  flag,  and  polarizability 
responses  obtained  by  a  single  dipole  inversion  performed  during  field  data  acquisition  were 
used.  We  developed  two  methods  for  selecting  a  single  template  position  to  represent  the 
anomaly.  The  first  method  selected  the  template  position  where  the  anomaly  was  estimated  to  be 
closest  to  the  sensor,  hereafter  referred  to  as  the  ‘closest’  method.  That  is,  the  inversion  estimate 
of  the  horizontal  distance  from  the  anomaly  to  the  sensor  was  smallest.  If  the  flag  location  were 
correct,  we  expected  the  central  template  position  (#2),  to  be  selected  most  frequently.  However, 
our  results  showed  that  template  positions  #1  and  #4  were  selected  more  frequently  than  the 
template  position  #2  (Figure  8b).  This  suggests  that  the  flag  locations  relative  to  the  anomaly 
were  frequently  outside  the  range  of  a  single  sensor  sounding.  The  second  method  was  to  select 
the  template  position  whose  anomaly  was  most  symmetric  along  the  transverse  axes,  hereafter 
referred  to  as  the  ‘symmetric’  method.  We  calculated  symmetry  using  the  mean  squared 
difference  between  log  polarizabilities  for  the  2nd  and  3rd  axes: 


where  i  G  {1, ...  5}  is  the  template  position,  p  is  the  log  polarizability  at  time  t  G  {1, ...  T}  for  the 
transverse  axes  y  and  z  (Figure  8a). 
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Template  Position  Template  Position 


(a)  (b) 

Figure  8.  Histogram  of  the  selected  template  position  for  the  sensor  across  all  anomalies  in 
BealeTrees  for  the  (a)  ‘symmetric’  and  (b)  ‘closest’  methods.  Template  position  #2  is  the  central 
template  position. 

Having  selected  a  single  template  position  for  each  flag  using  these  two  methods,  training  began 
by  selecting  an  initial  basis  of  30  vectors  maximizing  Fisher  Information  Gain.  This  initial 
selection  was  the  same  for  both  the  ‘closest’  and  ‘symmetric’  methods.  From  this  initial  sample  a 
set  of  relevant  features  was  selected  using  BENet,  and  a  non-linear  PNBC  classifier  trained  on 
these  features.  Then,  a  new  set  of  10  unlabeled  anomalies  were  selected  using  batch  AL.  The 
labels  for  these  anomalies  were  requested  and  the  training  process  was  repeated  until  new  labels 
did  not  provide  a  substantial  amount  of  new  information.  The  total  number  of  training  rounds  for 
the  ‘closest’  was  11  and  the  total  number  for  the  ‘symmetric’  method  was  10  (Figure  9).  At  the 
end  of  training  there  were  many  more  features  that  were  relevant  for  discrimination  in  the 
‘closest’  method  than  the  ‘symmetric’  method.  This  is  likely  due  to  the  fact  that  the  ‘closest’ 
method  selected  the  wrong  template  position  to  represent  an  anomly  more  frequently  than  the 
‘symmetric’  method  did. 
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Figure  9.  Fisher  Information  Gain  through  all  training  rounds  for  the  (a)  ’symmetric’  and  (b) 
‘closest’  methods. 
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Figure  10.  Distribution  of  posteriors  for  the  (a)  'symmetric'  and  (b)  ‘closest’  methods  at  the  end 
of  training. 


At  the  end  of  training  the  posterior  probabilities  of  unlabeled  samples  (Figure  10)  showed  that 
the  ‘closest’  method  was  not  as  separable  as  the  ‘symmetric’  method  at  this  stage;  some 
predicted  probabilities  for  the  previously  labeled  UXO  were  less  than  0.5.  We  dug  unlabeled 
observations,  beginning  from  the  highest  posterior  probability,  and  proceeded  until  we  had  dug  a 
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total  of  30%  of  the  site  (including  training  data).  There  were  three  ‘can’t  analyze’  anomalies, 
whose  positions  were  too  far  from  the  sensor  to  be  reliable  (BE-0609,  BE-0879,  and  BE-0887). 
After  receiving  the  labels  (ground  truth)  for  this  first  round  we  missed  eight  seeds  in  the  ‘closest’ 
method  and  three  seeds  in  the  ‘symmetric’  method.  Given  these  missed  seeds  we  retrained  and 
requested  43  additional  labels  for  the  ‘closest’  method,  and  eight  additional  labels  for  the 
‘symmetric’  method.  After  receiving  those  labels,  we  decided  to  stop  digging  the  ‘closest’ 
method,  but  continued  digging  the  ‘symmetric’  method  by  requesting  eight,  then  28,  and  finally 
22  additional  labels.  The  total  number  of  ranked  anomalies  (those  dug  but  not  counted  in  the 
training  or  ‘can’t  analyze’  sets)  at  the  end  of  digging  was  208  for  the  ‘symmetric’  method  and 
184  for  the  ‘closest’  method.  The  decision  to  stop  digging  the  ‘closest’  method  was  made 
because  our  last  50  digs  were  clutter,  even  though  there  still  remained  20  anomalies  with  a 
posterior  probability  greater  than  0.5  (Figure  11).  In  retrospect,  digging  these  remaining 
anomalies  would  not  have  revealed  any  target  of  interest  (TOI). 


Posterior  probability  Posterior  probability 

(a)  (b) 

Figure  11.  Posteriors  distributions  at  the  end  of  digging  for  the  (a)  ’symmetric’  and  (b)  ’closest’ 
methods. 
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4.2. 6  Retrospective  Analysis 


We  missed  a  group  of  fuzes  that  are  fairly  similar  in  a  feature  space.  The  primary  reason  we 
missed  this  type  of  targets  lies  in  our  feature  selection  process.  Each  time  a  classifier  was  built, 
the  feature  space  was  pruned  to  only  those  features  that  were  linearly  discriminative  of  UXO  and 
clutter.  But,  the  set  of  features  that  was  discriminative  for  most  of  the  UXO  was  not  the  same  set 
of  features  that  was  discriminative  for  fuzes  (Figure  12).  A  solution  to  this  problem  would  be  to 
create  a  separate  classification  for  fuzes.  A  secondary  reason  for  missing  these  fuzes  was  that 
they  were  surrounded  by  clutter  in  the  feature  space.  Figure  13  shows  that  while  the  missed  fuzes 
are  clustered  along  the  primary  axes  of  feature  variation,  there  are  also  many  clutter  items  close 
to  them.  During  active  learning  we  received  labels  for  anomalies  close  to  this  cluster  of  fuzes, 
and  they  were  all  clutter.  So,  any  model  we  fit  to  those  labels  would  assign  a  low  probability  of 
being  UXO  to  the  fuzes.  Again,  this  could  be  remedied  by  using  features  that  were  uniquely 
discriminative  for  fuzes  rather  than  being  discriminative  of  the  entire  dataset  (as  in  Figure  13)  or 
all  UXO  combined  (as  in  our  BENet  feature  selection). 

In  terms  of  selection  algorithm  performance,  the  ‘symmetric’  algorithm  tended  to  have  fewer 
incorrectly  selected  template  positions  than  the  ‘closest’  method.  There  are  a  few  lines  of 
evidence  for  this.  The  first  is  based  on  the  variance  of  features  for  UXO,  which  was  lower  for  the 
‘symmetric’  method  than  the  ‘closest’  method. 
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Figure  12.  Feature  weights  from  BENet  on  all  labeled  anomalies  for  all  UXO  (blue)  and  fuzes 
only  (red). 


The  second  line  of  evidence  for  the  ineffectiveness  of  the  ‘closest’  method  is  the  spread  of  UXO 
in  the  two-dimensional  ordination  for  the  ‘closest’  method  (Figure  13).  There  are  noticeable 
UXO  outliers  in  the  dominant  feature  axes  suggesting  the  polarizability  responses  for  these  UXO 
are  substantially  different  from  the  mean  responses  of  UXO.  These  outliers  are  not  present  in  the 
‘symmetric’  method,  leading  us  to  believe  these  outliers  are  the  result  of  selecting  an  incorrect 
template  position  in  the  ‘closest’  method.  An  example  of  inappropriate  selection  is  shown  in 
Figure  14.  For  this  ISO  every  other  template  position  had  an  ‘UXO-like’  response  except  the  one 
chosen  by  the  ‘closest’  method.  The  reason  for  such  inappropriate  selection  is  straight-forward. 
There  is  a  piece  of  clutter  underneath  template  position  #4;  one  not  associated  with  the  flagged 
anomaly.  This  piece  of  clutter  is  closer  to  the  sensor  than  the  UXO  was  to  any  of  the  other 
template  positions,  so  the  ‘closest’  method  selected  it. 
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Figure  13.  MAP  estimates  (color  scale)  along  two-dimensions  of  a  non-linear  ordination  of  the 
anomaly  features  for  the  ‘closest’  method  (top)  and  the  ‘symmetric’  method  (bottom).  The 
anomalies  that  were  marked  as  ‘dig’  in  our  dig  list  are  shown  (UXO  in  white,  clutter  in  black). 
The  missed  fuzes  for  each  method  are  shown  by  red  circles. 
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Figure  14.  Polarizabilities  for  flag  BE-0671,  which  is  an  ISO.  The  'closest'  method  selected 
template  position  #4  (black  box),  while  all  other  responses  are  more  UXO-like. 
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4.3  Discrimination  Using  Template-match  Approach 


4.3.1  Data  Analysis 


As  described  in  Chapter  3,  data  were  collected  using  the  three  or  five-point  template  placed  at 
the  flag  location.  The  orientation  of  the  system  was  based  on  the  ease  of  access  and  the  hill  slope. 
Data  in  J  and  K  grids  were  collected  with  the  system  oriented  towards  NW  (310-320°),  in  R7-9 
and  S7-9  grids  towards  SSE  (140-170°),  and  S2-3  and  T2-3  towards  SSW  (190-210°)  (Figure 
15a).  The  hill  slope,  or  tilt  of  the  system,  varied  up  to  15-20°  (Figure  15b). 


(a) 


(b) 

Figure  15.  Histogram  of  (a)  system  orientation,  and  (b)  slope  during  the  survey. 


For  the  discrimination  we  selected  one  representative  response  per  flag,  based  on  the 
polarizabilities  magnitude  and  symmetry,  to  ensure  that  if  multiple  objects  were  near  the  flag  the 
response  of  the  most  dangerous  item  would  be  selected.  Figure  16  shows  that  the  distribution 
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was  more  or  less  even  among  four  template  positions.  The  template  position  #4  was  selected  less 
frequently  since  it  was  uphill  from  the  flag  and  therefore  further  away  from  the  object.  Only 
about  20%  of  the  flags  had  three  measurements,  thus  we  analyzed  more  than  4,000 
measurements. 


Figure  16.  Histogram  of  the  selected  template  position  for  the  responses  used  in  discrimination. 
Position  #2  is  the  central  template  position. 


(a) 


(b) 


Figure  17.  Histogram  of  (a)  estimated  and  (b)  true  object  depths. 
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Figure  17  shows  a  very  good  agreement  between  objects  depths  estimated  by  inversion  (a)  and 
true  objects  depths  (b),  and  that  less  than  5%  of  the  objects  were  deeper  than  0.3  m. 

While  running  multiple  objects  inversions  takes  a  long  time  and  therefore  it  is  impractical  in  the 
field,  single  object  inversions  are  very  fast  and  available  right  after  the  data  are  acquired. 
Therefore  we  have  been  developing  and  evaluating  a  discrimination  approach  that  could  be  used 
in  a  near-real  time  in  the  field.  This  template -match  approach  used  data  processed  in  the  field 
using  a  single  object  inversion,  and  compared  polarizability  curves  of  unknown  objects  to  the 
polarizabilities  of  the  representative  objects  in  the  library.  Any  single  object  with  a  polarizability 
index  value  larger  than  50%  of  that  of  37-mm  projectile  (smallest  UXO)  was  considered  as  a 
potential  object  of  interest.  Parameters  used  in  this  approach  were  (a)  the  similarity  of  major 
polarizability  curves,  (b)  the  similarity  of  medium  and  minor  polarizability  curves,  (c)  the 
polarizability  index,  and  (d)  the  flag’s  average  polarizability  index.  We  used  data  from  100  flags 
for  training,  which  was  about  12%  of  the  whole  dataset.  Training  data  were  used  to  verify 
whether  single  object  inversions  could  be  used  for  discrimination,  and  to  determine  the  operating 
point  at  which  is  safe  to  stop  digging.  The  ranked  anomaly  list  that  had  the  operating  point  set  at 
486  is  discussed  in  Chapter  5,  and  the  retrospective  analysis  is  provided  in  Chapter  4.3.2. 

4.3.2  Retrospective  Analysis 

Due  to  a  slight  misunderstanding  what  was  considered  as  TOI,  our  original  stop-digging  point 
was  too  conservative.  If  small  fuzes  were  not  considered  as  TOI,  we  could  have  stopped  much 
sooner  and  saved  additional  150  digs.  Additional  savings  could  be  achieved  if  digging  would  be 
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done  by  contiguous  grids.  This  digging  approach,  however,  needs  a  follow-up  analysis,  since 
only  data  from  one  contiguous  area  or  previously  excavated  areas  could  be  used  for  training, 
which  wasn’t  done  in  the  presented  approach.  Our  training  data  selection  and  distribution  among 
survey  grids  suggests  that  this  is  feasible. 

The  last  UXO  on  the  ranked  anomaly  list  was  81-mm  mortar  (BE-4).  Given  the  object  size  and 
depth,  the  measured  response  wasn’t  representative  of  this  object,  and  this  location  should  have 
been  classified  as  ‘can’t  analyze’.  Doing  so  would  save  50  clutter  digs. 

Two  types  of  fuzes  were  present  at  the  site  -  (a)  60-mm  fuzes  that  had  the  polarizability  index 
comparable  with  the  response  of  37-mm  projectile  and  were  correctly  identified  using  our 
original  approach,  and  (b)  50-mm  fuzes,  which  responses  were  much  smaller  and  we  missed  two 
of  them  using  the  same  approach.  If  this  approach  would  be  used  for  real-time  discrimination  in 
the  field,  and  these  small  fuzes  would  be  considered  as  TOIs,  we  could  include  their  response  to 
our  library  of  representative  responses,  and  redefine  criteria  for  setting  up  the  operating  point  at 
two  different  stages  -  (a)  after  training  data  or  (b)  after  digging  the  highest  ranked  anomalies 
reveal  such  item.  Our  training  data  contained  small  fuzes,  however,  they  were  from  flags  with 
multiple  objects.  In  this  case,  we  would  have  to  either  acquire  data  over  the  pit  or  run  a  multi¬ 
object  inversion  to  obtain  a  single  fuze  polarizability  response.  Even  if  acquiring  data  over  the  pit 
wouldn’t  be  possible,  and  we  did  not  include  it  to  the  library  during  the  training  data  stage,  our 
ranked  anomaly  list  had  the  remaining  three  single  small  fuzes  at  positions  18,  201  and  226. 
Thus  we  would  encounter  them  before  our  stop-digging  point,  and  at  this  point  it  would  be  still 
feasible  to  include  the  response  to  the  library  and  regenerate  the  ranked  anomaly  list. 


Camp  Beale  Demonstration  Report 


32 


April  2012 


5.  PERFORMANCE  ASSESSMENT 


For  the  hand-held  prototype  performance  evaluation  we  submitted  a  separate  ranked  anomaly  list 
for  each  of  the  approaches  described  in  Chapter  4.  Each  flag  was  assigned  the  rank,  category, 
and  dig/no-dig  decision.  The  lowest  rank  had  the  flags  which  either  ‘can’t  be  analyzed’  or  had 
the  high  probability  of  being  of  UXO,  then  those  labeled  as  ‘can’t  decide’,  and  the  highest  rank 
had  flags  that  had  a  high  probability  of  being  scrap.  The  category  was  either  training  data,  or  test 
data,  and  dig/no-dig  decision  had  values  either  one  or  zero.  All  locations  with  dig  equal  to  one 
were  dug. 

5.1  Statistical  Approach  Performance 

Both  the  ‘symmetric’  and  ‘closest’  methods  ended  digging  up  all  the  TOI  when  fuzes  were 
considered  clutter  (Figure  18).  The  ‘closest’  method  performed  better  than  the  ‘symmetric’ 
method  in  number  of  false  alarms  to  reach  the  last  UXO:  180  vs.  200.  We  would  consider  both 
methods  to  be  conservative  given  the  fact  that  we  dug  approximately  50  extra  clutter  items  after 
all  the  non-fuze  TOI  had  been  revealed,  and  given  that  we  acquired  over  100  training  labels 
using  AL  in  both  methods.  In  theory  this  means  that  we  thoroughly  explored  the  feature  space 
for  hidden  modes  of  TOI.  But,  the  ‘closest’  method  missed  three  fuzes  and  the  ‘symmetric’ 
method  missed  four  fuzes  when  these  items  were  considered  TOI.  Both  methods  missed  BE- 
0482,  BE-558,  and  BE-0805,  and  the  ‘symmetric’  method  additionally  missed  BE-0697.  The 
reasons  why  we  missed  these  fuzes  are  given  in  Chapter  4.2.6. 


Camp  Beale  Demonstration  Report 


33 


April  2012 


Percent  of  UXOs  Correctly  Classified  (%)  Percent  of  UXOs  Correctly  Classified  (%) 


(a)  (b) 


(c) 


(d) 


Figure  18.  ROC  curves  for  the  ‘symmetric’  method  (top)  and  the  ‘closest’  method  (bottom). 
Fuzes  are  treated  as  UXO  (left)  and  clutter  (right). 
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5.2  Template-match  Approach  Performance 


The  template -match  approach  used  the  criteria  described  in  Chapter  4.3  to  select  the  operating 
point  (dig  threshold),  and  Figure  19  illustrates  that  even  with  those  criteria  more  than  40%  of 
anomalies  were  correctly  labeled  as  non-dangerous  if  the  smallest  fuzes  were  not  considered  as 
TOIs.  This  operating  point  (blue  dot)  was  very  conservative,  and  the  retrospective  best  dig 
threshold  (yellow  dot)  would  save  another  150  digs.  If  the  anomalies  would  be  excavated  grid- 
by-grid,  additional  80  holes  would  be  saved  (magenta  dot).  We  missed  two  small  fuzes  (BE-558 
and  BE-0805),  if  they  were  considered  as  TOIs.  Our  retrospective  analysis  in  Chapter  4.3.2 
describes  possible  solutions,  and  suggests  that  this  approach  could  potentially  save  more  than 
60%  of  clutter  digs  if  the  operating  point  would  be  selected  iteratively  based  on  the  information 
from  digging  locations  that  were  high  on  the  ranked  anomaly  list. 


Figure  19.  The  template-match  approach  ROC  curve.  Fuzes  are  treated  as  clutter. 
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6.  CONCLUSIONS 


This  survey  showed  that  the  same  discrimination  capabilities  afforded  by  the  cart-mounted 
system  are  available  in  the  hand-held  unit,  although  with  a  slightly  reduced  depth  of  detection. 
The  survey  in  a  steep  terrain  in  wooded  areas  was  challenging  but  feasible.  While  the  system  is 
operable  by  a  single  operator,  because  it  has  no  positioning  navigation,  the  second  person  was 
necessary  to  ensure  the  system  was  positioned  correctly  at  the  template  positions.  We  had  no 
downtime,  collected  over  4,000  measurements,  and  finished  the  survey  in  two  weeks.  The 
discrimination  results  were  excellent  -  only  25%  of  the  anomalies  had  to  be  excavated  to  find 
every  TOI.  Moreover,  our  near-real  time  discrimination  approach  also  showed  promising  results. 
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9.  ACRONYMS 


AL 

Active  Learning 

BENet 

Bayesian  Elastic  Net 

BUD 

Berkeley  UXO  Discriminator 

ESTCP 

Environmental  Security  Technology  Certification  Program 

FPGA 

Field  Programmable  Gate  Array 

IDA 

Institute  for  Defense  Analyses 

LBNL 

Lawrence  Berkeley  National  Laboratory 

MAP 

Maximum  a  posteriori 

MTL 

Multi-task  learning 

PNBC 

Parametric  Neighborhood-Based  Classifier 

ROC 

Receiver  Operating  Characteristic 

SERDP 

Strategic  Environmental  Research  and  Development  Program 

SIG 

Signal  Innovations  Group,  Inc. 

S/N 

Signal-to-noise 

TOI 

Target  of  Interest 

UXO 

Unexploded  Ordnance 
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